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Description 

RESPONSIVE ENVIRONMENT SENSOR 
SYSTEMS WITH DELAYED ACTIVATION 

Cross Reference to Related Applications 

[0001] This application claims priority under 35 U.S.C. section 
119(e) from Provisional Patent Application Serial Number 
60/521,613, filed June 6, 2004, entitled Responsive Envi- 
ronment Sensor Systems With Delayed Activation (Attorney 
Docket Number F-822), which is incorporated herein by 
reference in its entirety.This application claims priority 
under 35 U.S.C. section 119(e) from Provisional Patent 
Application Serial Number 60/521,747, filed June 6, 2004, 
entitled Responsive Environment (Attorney Docket Num- 
ber F-822a), which is incorporated herein by reference in 

its entirety. 
Background of Invention 

[0002] The illustrative embodiments described in the present ap- 
plication are useful in systems including those for use in 
responsive environments and more particularly are useful 



in systems including tliose for application and function 
activation in a responsive environment. 

[0003] The term responsive environment may be used to describe 
an environment that has computing capability and access 
to sensing technology data that allows the environment 
control to consider its current state or context and new 
events that occur that may change the state or context. 

[0004] Some examples of prototype responsive environments in- 
clude the BlueSpace system described by IBM Corporation 
of White Plains, New York, the Stanford Iroom system de- 
scribed by Stanford University of Stanford, California and 
the Aware Home system described by Georgia Institute of 
Technology of Atlanta, Georgia. 

[0005] Tvvo types of responsiveness are traditionally supported in 
responsive environments. In the first type, an application 
currently running in the environment receives an event 
message including sensed data. The application then per- 
forms a function that is determined by the sensed data 
that represents the sensed event. For example, an envi- 
ronment management application may be running in an 
office. The application may be programmed to set the 
lights, temperature, and white noise level of the office 
when a person enters it by using information regarding 



the identity of tlie person and tlie time that the person 
entered the office. In this example, the sensed event is 
that someone entered the office and it includes the iden- 
tity of the entering person. 

[0006] In a second type of responsiveness, environment func- 
tionality may be enhanced based on sensed events. Cer- 
tain applications that were not previously running could 
be dynamically started in response to a sensed event. Al- 
ternatively, certain applications that were running could 
be terminated due to a sensed event. For example, a 
meeting room may automatically launch a presentation 
application to support a meeting that is about to begin 
when the presenter enters the room. 

[0007] Several groups have described systems related to respon- 
sive environments and context aware computing. In a first 
example, a project named the Context Toolkit is de- 
scribed in relation to the Aware Home system in a paper 
entitled "A Conceptual Framework and a Toolkit for Sup- 
porting the Rapid Prototyping of Context-Aware Applica- 
tions"by Anind K. Dey, Daniel Salber and Gregory D. 
Abowd as found in the Human-Computer Interaction (HCI) 
Journal, Volume 16 (2-4), 2001, pp. 97-166. 

[0008] In a second and third example, the Iroom system is de- 



scribed. In the second example, a group described the 
Event Heap system in a paper entitled "The Event Heap: A 
Coordination Infrastructure for Interactive Workspaces" by 
Brad Johanson and Armando Fox as found in the Proceed- 
ings of the Fourth IEEE Workshop on Mobile Computing 
Systems and Applications, p. 83, June 20-21, 2002. 

[0009] In the third example, a mechanism to introduce mediation 
into decisions initiated by a responsive environment is de- 
scribed in a paper entitled "Distributed Mediation of Am- 
biguous Context in Aware Environments" by Anind K. Dey, 
Jennifer Mankoff, Gregory D. Abowd and Scott Carter as 
found in the Proceedings of the 15th Annual Symposium 
on User Interface Software and Technology (UlST 2002), 
Paris, France, October 28-30, 2002. pp. 121-130. 

[0010] The first two examples describe mechanisms for building 
responsive environments. In the first example, applica- 
tions within an environment are made responsive by di- 
rectly building in their reactions to sensed events during 
the design of the application. In the second example, 
sensed events are all published to a well-known space, so 
that applications can access them easily at run time. Ac- 
cordingly, those applications can be responsive to those 
events. For both the first and second examples, the appli- 



cation designers specifically build in responsiveness. The 
applications execute this responsiveness directly in a pre- 
determined fashion, whether or not the user wants it. 

[0011] The system described in the third example apparently 
provides an attempt at mediation to better handle errors 
of decisions that the environment may make. It introduces 
special mediation components that, given that the envi- 
ronment is responding, can reduce the error in the deci- 
sion that the environment makes. For example, an appli- 
cation in the environment may try to support the user with 
word completion such that the system guesses the rest of 
a word being typed after only a few letters are typed. 
Given the context, the environment may complete words 
differently. The mediation components allow users to 
more specifically say what the relevant context is, so that 
the decisions made by the applications are more support- 
ive. However, the mechanism does not allow the user to 
control when the environment initiates the decision pro- 
cess. Rather, it assumes that the response will happen and 
the system merely allows the user to mediate the re- 
sponse to provide better decisions. 

[0012] Traditional responsive environments are very much hard- 
wired. That is, a sensor is tied to an application, and when 



it senses an event, it immediately affects the behavior of 
that application in a predetermined way. There is little or 
no opportunity for user interaction in this type of system. 
In most cases, the level of functionality is pre-set and new 
applications are not typically launched. 
[0013] However, in those traditional responsive environments 
that are able to launch other applications, they do so di- 
rectly and immediately based on predetermined sensed 
events. Users might be able to initially configure which 
event causes which application to startup, but when the 
event happens, the designated application starts automat- 
ically. 

[0014] Accordingly, among other things, the prior art does not 

provide a responsive environment having delayed applica- 
tion and/or function activation. 
Summary of Invention 

[0015] The illustrative embodiments described herein overcome 
the disadvantages of the prior art by providing a method 
and system for delayed function activation in a responsive 
environment. In one illustrative embodiment, a responsive 
environment includes a notification manager intermediary 
system used to determine when a responsive environment 
will launch an application triggered by a sensed event. The 



environment includes sensor components and transparent 
software components that reside wholly within the envi- 
ronment for providing responsiveness. The environment 
also includes interactive software components that give 
users the means to provide input and/or experience out- 
put. 

[0016] jhe transparent components build dynamic context within 
the environment, determine if the environment context 
has changed, and if the context has changed, cause the 
environment to respond. Such response may include initi- 
ating a functional and/or content change in an active in- 
teractive component. Additionally, the response may in- 
clude starting and/or stopping one or more components 
or applications. Since changes to the interaction compo- 
nents may disrupt the user by requiring the user"s atten- 
tion when it is not convenient, the system helps the user 
manage the responsive environment by using an applica- 
tion/function delay technique. 

[0017] The environment uses intelligence to determine what 

might be useful and primes the apparatus to provide that 
functionality. However, the user actually initiates the exe- 
cution of that function. The system utilizes a notification 
mechanism along with the primed apparatus as the means 



to trigger the function. 
Brief Description of Drawings 



[0018] FIG. 1 is a scliematic representation of a representative 
responsive environment according to an illustrative em- 
bodiment of the present application. 

[0019] FIG. 2 is a schematic representation of the response to a 
change of state of a representative responsive environ- 
ment according to an illustrative embodiment of the 
present application. 

[0020] FIG. 3 is a schematic representation of the response to a 
change of state of another representative responsive envi- 
ronment according to an illustrative embodiment of the 
present application. 

[0021] FIG. 4 is a flowchart showing a representative method for 
using a notification manager system according to an illus- 
trative embodiment of the present application. 

[0022] FIG. 5 is a schematic representation of a representative 
notification interface browser used with a responsive en- 
vironment according to an illustrative embodiment of the 
present application. 

[0023] FIG. 6 is a flowchart showing a representative method for 
using a notification manager system according to an illus- 
trative embodiment of the present application. 



Detailed Description 

[0024] Illustrative embodiments describing systems and metliods 
for responsive environments having delayed application 
and/or function activation are described. The illustrative 
embodiments are described with reference to particular 
representative configurations, sensors and data types, but 
systems according to the present application may use var- 
ied and variable configurations, sensor types and data and 
functionality types. 

[0025] Several disadvantages of traditional responsive environ- 
ments exist. Users may feel that responsive environments 
are just too responsive. The occupants of a room may 
want more fine-grained control. For example, the occu- 
pant may want the lights dimmed, but not quite yet. The 
occupant might want the presentation software to start, 
but not until after introductions have occurred. 

[0026] Context is defined to comprise information and in at least 
some embodiments includes two sets of information. The 
first set of information includes statistically registered en- 
tities of interest, each with a predefined set of character- 
istics. Entities of interest include people, devices, applica- 
tions, sensors, spaces, tasks and documents. Characteris- 
tics are based upon the type of entity. For example, docu- 



ment characteristics may include among otiier things a ti- 
tle and a date of creation. The characteristics of people 
may include for example a full name and a nickname. The 
second set of information includes dynamic relationships 
between (among) the entities of interest. For example, a 
person is using a projector and a document is present in 
the room. In that situation, there are at least two types of 
relationships to illustrate. There is a relationship between 
the people and devices and there is a relationship be- 
tween the documents and the spaces. 

[0027] Many people do not like notifications or windows sud- 
denly popping up on their computer display screens while 
they are working with a particular application such as a 
word processor. Such interruptions disrupt their work. Ac- 
cordingly a system was needed that could have the envi- 
ronment support the user's work and at the same time re- 
duce its intrusiveness. 

[0028] The illustrative embodiments of the present application 
provide several advantages over prior systems. First, the 
embodiments describe responsive environments that 
manage the context and that can use that context intelli- 
gently to support the users of the environment. Tradi- 
tional systems focused instead on having the application 



react to the environment. Secondly, the embodiments de- 
scribe responsive environments in which users have con- 
trol over when and how the recommended function is 
done. Thirdly, the embodiments describe responsive envi- 
ronments in which user attention is better managed. For 
example, the responsive environment does not force a 
user to do things on a particular predetermined timetable. 
Instead, the user can do things when it is convenient. Ad- 
ditional advantages will be apparent as the embodiments 
are described. 

[0029] Accordingly, the illustrative embodiments described focus 
on a method and supporting apparatus for enhancing the 
functionality of a responsive environment that gives users 
control over the two traditionally supported types of re- 
sponsiveness described above using a single mechanism. 
Furthermore, the embodiments described provide this 
control selectively such that the method and apparatus is 
non-disruptive to the existing environment. Accordingly, 
the systems described may co-exist with the traditional 
systems that support responsiveness. The illustrative em- 
bodiments leverage the available intelligence of the re- 
sponsive environment to determine the range of behaviors 
a user may want to happen, given the context of the envi- 



ronment. However, the embodiments described also pro- 
vide a meclianism to delay the responsiveness in a way 
that allows the user to have more control. In addition, the 
systems do so in a way that is not intrusive to the user. 
[0030] In one embodiment, a responsive environment is instru- 
mented with sensors, transparent software components 
that reside wholly within the environment for providing 
responsiveness, and interactive software components that 
give users the means to provide input and/or experience 
output. The sensor components provide sensor data. The 
transparent components build dynamic context within the 
environment, determine if the environment context has 
changed, and if the context has changed, cause the envi- 
ronment to respond, often by initiating a functional/con- 
tent change in an active interactive component or by 
starting and/or stopping one or more components. 
Changes to the interaction components may disrupt the 
user, requiring his/her attention when not available. In 
this embodiment, the system helps the user manage his/ 
her attention within a responsive environment by using an 
application/function delay technique. The environment 
uses it's intelligence to determine what might be useful 
and primes the apparatus to provide that functionality. 



but the user is the one to actually initiate the execution of 
that function. The invention uses a specialized notification 
mechanism, tied to the primed apparatus, as the means to 
trigger the function. 

[0031] Referring to FIG. 1, an illustrative responsive environment 
10 according to an illustrative embodiment of the present 
application is shown. The representative responsive envi- 
ronment has been implemented in a system known as 
Atira that includes a context management infrastructure 
that includes a layered framework of incremental intelli- 
gence in the form of a PAUR pyramid 20 that has four lay- 
ers each including components that have similar overall 
roles. The components pass messages up to the layer 
above. However, different components in a particular layer 
may provide specialized functionality by subscribing to a 
subset of messages from the layer below. 

[0032] External stimuli are sensed using physical or logical sen- 
sors 31, 33, 35 and 37. The stimuli enter the pyramid 2 
through sensor/trigger components 32, 34, 36, 38 that 
interact directly with the sensors. Those triggers typically 
only publish into the pyramid rather than subscribe to 
messages. The lowest layer of the pyramid is the P or Per- 
ception layer 28 and it includes several perception com- 



ponents 42, 44. The perception components may sub- 
scribe to stimuli events. Similarly, the perception compo- 
nents may publish to the next higher level. The Perceptors 
are used to filter the types of external stimuli that are 
used to build the context. 
[0033] The next level of the pyramid 20 is the A - Awareness 
layer 26. The awareness layer components 52, 54 are 
known as Monitors. The monitors manage the state of ac- 
tive entities that are known in the context such as docu- 
ment, application or task entities. The monitors 52, 54 
manage the overall state of the environment by updating 
properties associated with entities. They determine the 
occurrence of activities such as a person carrying a partic- 
ular document that may also indicate an additional change 
in state. They also manage the relationships among the 
entities. 

[0034] The next level of the pyramid 20 is the U - Understanding 
layer 24. The understanding layer components 62, 64 are 
known as Grokkers. The grokkers determine the types of 
activities that are underway in the environment. The 
grokkers determine if changes in the context merit a 
change in behavior in the room, and if so, determines the 
type of behavior and initiates it. Grokkers are also utilized 



to prime applications. 

[0035] The final level of the pyramid 20 is the R - Response layer 
22. The response layer components 72, 74 are known as 
Responders. The responders semantically drive the envi- 
ronment function and prepare and deliver an announce- 
ment that describes the needed behavior. The applications 
in the environment use the announcements to decide if 
any function is needed. 

[0036] The responsive environment 10 includes thin client appli- 
cations that reside outside of the context infrastructure 
30. For example, an interface browser application 80 may 
be used to view objects in the environment. Additionally, 
an application launcher client 82 may be used to launch 
external applications based upon the context contained in 
the PAUR pyramid 20. A Notification Manager can be a 
thin client application with an interactive component that 
manages the user"s attention. For example, the thin 
clients 80, 82 include actuators 86 and 88 that are part of 
the thin client systems. The actuators and thin clients may 
subscribe to announcements of the system and can also 
include triggers to create internal stimuli such as an ap- 
plication-entered environment. 

[0037] The illustrative responsive environment system described 



utilizes a central server computing system comprising one 
or more DELL®servers having an IN- 
TEL®PENTIUM®processor running the WINDOWS® XP oper- 
ating system. The system is programmed using the JBOSS 
system and the Java Messaging System QMS) provides the 
publish/subscribe messaging system used in the respon- 
sive environment. 
[0038] In an illustrative embodiment, physical sensor 31 is a 

scanner system that also includes a computer that inter- 
faces with the sensor component 32 using a serial line or 
TCP/IP interface. The connections among the physical 
systems that comprise the logical system 90 include wire- 
less and wired connections among physical computers 
running the appropriate applications, components and 
frameworks. Sensors 35, 37 are RFID sensors each includ- 
ing a computer that interfaces with the respective sensor 
components using a serial line. Sensor 33 may comprise 
well-known sensors such as thermometers, pressure sen- 
sors, odor sensors, noise sensors, motion sensors, light 
sensors, passive infrared sensors and other well-known 
sensors. Additional well-known communications channels 
may also be used. In the illustrative embodiment de- 
scribed, the JBOSS JMS message space is running on one 



server while the MySQL system is run using another server 
to maintain tables used in the RDF system. Additionally, 
the PAUR components such as component 42 are all run- 
ning on a third server. The thin clients 80, 82 and thin 
client components 86, 88 are running on separate client 
machines in communication with the system 90. 

[0039] jhe responsive environment described herein is illustra- 
tive and other systems may also be used. For example, a 
querying infrastructure could be used in place of the noti- 
fication or publish/subscribe system that is described 
above. Similarly, the messaging service could be provided 
across systems and even across diverse system architec- 
tures using appropriate translators. While IN- 
TEL®processor based systems are described using MI- 
CROSOFT ®WINDOWS systems, other processors and op- 
erating systems such as those available from Sun Mi- 
crosystems may be utilized. 

[0040] Referring to FIG. 2, the response to a change of state of a 
representative responsive environment 200 according to 
an illustrative embodiment of the present application is 
shown. 

[0041] Responsive environments typically include systems for 
managing the state of the environment, sensing events 



that may change that state and driving responses to the 
change in state. In this embodiment, the state is repre- 
sented using a series of models. The models include a set 
of entity models and a model that contains relationships 
among the entities as shown in FIG. 2. All of the models 
shown in this embodiment are described using Resource 
Description Framework (RDF) descriptions and are per- 
sisted using a relational database. The Jena system avail- 
able from the Hewlett-Packard Company of Palo Alto, Cal- 
ifornia is used as an RDF manager, but the OWL ontology 
being standardized by the World Wide Web consortium 
may also be utilized. 
[0042] The environment 200 supports several different types of 
entities, as shown by the plurality of representative entity 
models in the state of the environment 210. All of the en- 
tities are uniquely identified using a Uniform Resource In- 
dicator (URI). These entities may include persons 214, de- 
vices 216, spaces 218, documents 220, and applications 
222. Additionally, a model for the relationships among the 
entities is maintained 212. Furthermore, task entities may 
be maintained. 

[0043] When a change in state 230 occurs, a message or event is 
generated and posted to the environment. The environ- 



ment 200 includes special components named Responsive 
Agents 240 that receive the messages or events that cor- 
respond to the grokkers. The responsive agent 240 then 
determines the types of function and/or activities that the 
environment should perform as a result of that change in 
state. The responsive agent 240 then communicates di- 
rectly with the affected Environment Services 250, 252 to 
have the service carry out the function. 
[0044] Referring to FIG. 3, the response to a change of state of a 
representative responsive environment 300 according to 
an illustrative embodiment of the present application is 
shown. 

[0045] In this embodiment, an intermediary named the Notifica- 
tion Manager 360 is provided. Additionally, the state of 
the environment 310 models also includes a Notification 
Model 324. The environment 300 also supports several 
different types of entities that are also uniquely identified 
using a Uniform Resource Indicator (URI). These entities 
may include persons 314, devices 316, spaces 318, docu- 
ments 320, and applications 322. Additionally, a model 
for the relationships among the entities is maintained 
312. 

[0046] When a change in state 330 occurs in this embodiment. 



the Responsive Agent 340 determines tlie type of func- 
tion/activity that the environment should provide in re- 
sponse to that change. However, rather than communicat- 
ing directly with the Environment Service and Activity 
components 350, 352, the Agent 340 first defines de- 
scriptions 344 for the needed application instances in the 
application model 322 and stores the needed inputs in 
that description. The Agent 340 then creates a new notifi- 
cation 342, defines its description in the notification 
model 324, associating the newly created application in- 
stance descriptions within the notification description. It 
then passes the notification to the Notification Manager 
360. 

[0047] Referring to FIG. 4, a flowchart describing a representative 
process 400 for using a notification manager system ac- 
cording to an illustrative embodiment of the present ap- 
plication is shown. 

[0048] Initially, in step 410, the process determines the appro- 
priate response, given the state change. In step 420, the 
process creates application instance descriptions, storing 
any inputs needed to actually instantiate the application. 
In step 430, the process creates a notification description 
linking the URIs to the previously created application in- 



stance descriptions. Finally, in step 440, the process in- 
forms the notification manager of the new notification. 

[0049] Here, the application instance description includes a link 
to a description in the application model that describes 
the type of application that should be started. When the 
appropriate response includes augmenting the environ- 
ment with a new application service/activity, the linked 
application instance description describes that service/ac- 
tivity. When the response is to have an existing applica- 
tion service/activity execute function, the linked applica- 
tion instance is a special mediator application that allows 
the user to customize parameters for that function and 
that then communicates with the targeted service/activity 
to initiate the function. 

[0050] Referring to FIG. 5, a representative notification interface 
browser 500 used with a responsive environment accord- 
ing to an illustrative embodiment of the present applica- 
tion is shown. 

[0051] The Notification Manager is implemented with the envi- 
ronment browser application 80. The environment 
browser 500 provides users of the environment away of 
viewing private and public context within the environment 
such as the presence of a new folder 520 that is awaiting 



a label input from the user. Accordingly, it also provides 
an excellent place to send notification messages. 

[0052] The notifications of the illustrative embodiment described 
here utilize a traditional email mechanism. However, other 
known notification systems may be used. If a new notifi- 
cation is received, then a non-intrusive icon 510 is shown 
at the bottom of the window. The user can browse to view 
the notifications, as listed on the My Notices panel 530, at 
his/her own convenience. 

[0053] Referring to FIG. 6, a flowchart describing a representative 
process 600 for using a notification manager system ac- 
cording to an illustrative embodiment of the present ap- 
plication is shown. 

[0054] When the user selects a notification in the panel, the fol- 
lowing steps occur. In step 610, the Notification Manager 
reports to the environment that the user has selected the 
notification since that action may cause the environment 
to change context and thus behavior. For example, the 
initiator of the initial message may have requested to be 
notified when the recipient clearly received the message. 

[0055] In step 620, the Notification Manager then accesses the 
description for the selected notification object from the 
notification model. The process also displays the notifica- 



tion information to tlie user. 

[0056] In step 630, the system retrieves tlie URI for tlie applica- 
tion instance tliat it needs to start, and uses tliat to re- 
trieve tlie description of tlie application instance. 

[0057] If the environment browser is not currently active a notifi- 
cation icon is sent to the user"s task bar. The taskbar icon 
opens the environment browser application and the user 
can then access notifications as described above. 

[0058] In step 640, the system displays information about the 
notification to the user and application parameters and 
queries the user to allow the user the opportunity to reject 
it. By rejecting it, any enhanced functionality will not occur 
and the process will end in step 690. Note that instead of 
rejecting the notification, the user may modify some of 
the application parameters, if desired. Additionally, the 
user may decide to further delay the function. The accep- 
tance or rejection may be implicit by allowing a timer to 
time out or may be explicit by having the user perform a 
selection operation. 

[0059] In step 650, if the notification has not been rejected, the 
manager starts the application, using the parameters as 
modified by the user. 

[0060] The illustrative embodiments described herein provide a 



common apparatus for delaying responsiveness within a 
responsive environment until a user has the desire to al- 
low the responsiveness or until the user has the time to 
control/modify some aspect of the responsiveness. 
[0061] If the responsive environment determines that new appli- 
cations should be launched, the new applications will be 
launched as soon as the user actively acknowledges/ac- 
cepts the associated notification. If the responsive envi- 
ronment determines that an existing application should 
change behavior, the application started by the apparatus 
is a mediator that the user can use to modify/customize 
the response. 

[0062] This systems described may coexist with traditional sys- 
tems that employ only automatic responsive functionally. 
The Responsive Agents may leverage the new functionality 
or may execute in a more traditional manner. 

[0063] Co-pending, commonly owned U.S. Patent Application 
No.: 10/176,248, filed June 19, 2002, is entitled Method 
and system for creation and use of webs of linked docu- 
ments (attorney docket no. F-326) and is incorporated 
herein by reference in its entirety. 

[0064] Co-pending, commonly owned U.S. Patent Application 
No.: TBD, filed on even date herewith, is entitled Method 



and System For Determining Location By Implication 
(attorney docket no. F-871) and is incorporated herein by 
reference in its entirety. 
[0065] Co-pending, commonly owned U.S. Patent Application 
No.: TBD, filed on even date herewith, is entitled Method 
and System For Deployment of Sensors (attorney docket 
no. F-870) and is incorporated herein by reference in its 
entirety. 

[0066] The present application describes illustrative embodi- 
ments of a system and method for delayed function acti- 
vation in a responsive environment. The embodiments are 
illustrative and not intended to present an exhaustive list 
of possible configurations. Where alternative elements are 
described, they are understood to fully describe alterna- 
tive embodiments without repeating common elements 
whether or not expressly stated to so relate. Similarly, al- 
ternatives described for elements used in more than one 
embodiment are understood to describe alternative em- 
bodiments for each of the described embodiments having 
that element. 

[0067] The described embodiments are illustrative and the above 
description may indicate to those skilled in the art addi- 
tional ways in which the principles of this invention may 



be used without departing from tlie spirit of the invention. 
Accordingly, the scope of each of the claims is not to be 
limited by the particular embodiments described. 



